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ABSTRACT 



This thesis addresses the Test and Evaluation process and 
demonstrates, through application of a dynamic systems model, 
that the Test and Evaluation process is a conglomerate of 
lower-order systems. Each system can be characteristically 
analyzed by focusing on objectives, inputs/outputs, interfaces, 
environmental, and effectiveness dimensions. Hence the systems 
approach to management is suggested as a paradigm for the Test 
and Evaluation process. This approach provides a systematic 
way of thinking about the job of management and emphasizes 
viewing the Test and Evaluation system as a whole instead of 
as segregated parts. The classic functions of management can 
be put in juxtaposition with the phases of Test and Evaluation 
to provide a better -insight to management of scarce resourses. 
Implementation of a strategy is perhaps the most important 
first step in practically using the systems approach and as 
such stimulates the manager to think in the proper perspective. 
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INTRODUCTION 



A. BACKGROUND 

Since the mid 1960 's all phases of the weapon systems 
acquisition process within the Department of Defense (DoD) 
have undergone criticism and change. Excessive life cycle 
cost growth and poor operational performance of weapon systems 
upon entering use have been frequent problems. As a result, 
several committees were formed to extensively study these 
problems; among them the President's Blue Ribbon Defense Panel, 
a General Accounting Office committee, and the Congress's 
Commission on Government Procurement. All have consistently 
identified Test and Evaluation as a major problem area. 

The acquisition policy for major weapon systems has evolved 
from the high risk "total package procurement" philosophy of 
development under a fixed-price contract to the more conserva- 
tive "fly-before-buy" concept of developing a new system under 
hybrid cost-type contracts. Under the latter concept program 
risks and progress are assessed at specific milestone events 
prior to proceeding into production. Concurrency between 
development and production is being minimized and greater 
emphasis is placed on early determination of operational suit- 
ability prior to production commitments . This change in 
system acquisition policy has thus placed greater emphasis 
on the Test and Evaluation process. 
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The overall Navy Test and Evaluation (T&E) process can 
be viewed as a system composed of testing agencies interacting 
in such a manner as to test and evaluate a weapon system. This 
process is composed basically of three types of T&E; Development 
Test and Evaluation (DT&E) , Operational Test and Evaluation 
(OT&E) , and Production Acceptance Test and Evaluation (PAT&E) . 
Each type consists of a heirarchy of several layers of organi- 
zational systems; from Secretary of the Navy CSecNav) down to 
the individual field activities under the Naval Air Systems 
Command (NAVAIRSYSCOM) , or to the squadrons under the Operations 
Test and Evaluation Force (OPTEVFOR) . Each of the field ac- 
tivities can in turn be subdivided into various systems; from 
the Directorate level down to the Section level. Each of these 
layers of systems within the types of T&E performed can be 
thought of as parts or elements within the total T&E process. 

The T&E process within DoD is characteristically bureaucratic. 
Chief of Naval Operations (CNO) Instruction 3960.10/ DoD 
Directive 5000.3 and SecNav Instruction 5000.1, among others, 
have established the WHO , WHAT , WHY , and WHEN of the Navy T&E 
process. However the question of HOW to effectively manage 
T&E remains outstanding and must be addressed. If history is 
any indication of the future the T&E process will become 
steadily more complex as the environment about the weapons 
acquisition process becomes more dynamic, as scarce resourses 
become more scarce, and as weapon systems become more complex. 

The question of HOW to manage the T&E process of the future 
becomes a perplexing and important issue. The manager of the 
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future must be able to cope with the ever changing dynamic 
nature of the environment about him and also with the com- 
plexities of the many organizational systems which comprise 
the total T&E system process. This thesis suggests a manage- 
rial approach which could cope with these complexities. This 
approach, the systems approach to management, can provide a 
framework for thinking which guides the manager of any organi- 
zational system to consider the objectives, environment, inputs 
and outputs, and interfaces that affect that system; and more 
importantly, to view the T&E system as a whole instead of as 
fragmented parts. 

B. OBJECTIVES OF THESIS 

The primary objectives of this thesis are two fold. First 
is to show that the Test and Evaluation process is in fact a 
system composed of many interrelated organizational systems. 

The second objective is to show that the systems approach to 
management offers a framework for managing the Test and Evalua- 
tion process of the future and as such becomes a useful manage- 
rial tool. 

C. SCOPE OF THESIS 

The systems approach to management can apply to any system 
or subsystem within the total Test and Evaluation process. 

This thesis is however restricted in scope to the Navy Test 
and Evaluation process within the Department of Defense. This 
thesis will draw from managerial experience gained during the 
Test and Evaluation process of weapon systems, i.e. F-14 et al 
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and relate the future managerial complexities of T&E to the 
systems approach of management. The overall concepts are 
ceraintly applicable however to T&E in general. 



D . METHODOLOGY 

This study is comprised of five basic phases, the last four 
of which overlap to some extent, but are nevertheless distinctly 
different. 



Phase I. 


Identification of the problem and establish- 
ment of basic objectives. 


Phase II. 


Literature search in the areas of Systems 
Management, Program Management, Test and 
Evaluation, Government manuals, specifications 
and directives. 


Phase III. 


Information gathering from various agencies. 
Appendix A presents organizations inter- 

viewed for information pertaining to this 
thesis . 


Phase IV. 


Analysis of information. 


Phase V. 


Preparation of thesis. 



The process of analyzing Test and Evaluation begins by first 
analyzing the systems approach to management. Chapter II pre- 
sents the concepts and characteristics of the systems approach 
and formulates a working systems model which can be applied 
throughout this thesis. 

Chapter III analyzes the Test and Evaluation process and 
policies and delineates the basic types of test and evaluation 
performed on a weapon system. Chapter IV analyzes the separate 
types of T&E from a systems view applying the systems model 
presented in Chapter II. Chapter V analyzes the total T&E 
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process from a systems view. Chapter VI concludes the text by 
relating the systems approach to the management functions 
within the T&E process and suggests the importance of strategy 
formulation and implementation. Conclusions then summarize 
the basic findings of this thesis. 
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II. THE SYSTEMS APPROACH TO MANAGEMENT 



The systems approach to management provides a systematic 
framework of thinking about the job of management and stresses 
the interrelatedness and interdependency of the parts to the 
whole. A systems approach has long been observed by scholars 
and practitioners alike. One of the earliest recorded state- 
ments concerning systems and systems management can be traced 
to 500 B.C. when Mencius stated: 

"Whoever pursues a business in this world must have a 
system. A business which has attained success without 
a system does not exist. From ministers and generals 
down to the hundreds of craftsmen, everyone of them, 
both skilled and unskilled, use this system. The skilled 
may at times accomplish a circle and a square by their 
own dexterity. But with a system, even the unskilled 
may achieve the same results, though dexterity they have 
none. Hence, every craftsman possesses a system as a 
model. Now, if we govern the empire, or a large state, 
without a system as a model, are we not even less in- 
telligent than a common craftsman?" [George, 1970] 



However, only since the mid 1960 's has the systems approach 
evolved into the managerial area. Appendix B presents the 
evolution of management concepts and places the systems approach, 
of management in perspective with other classical theories. 



A. THE CONCEPT OF A SYSTEM 

A Starting point in the analysis of the systems approach 

is first to define what a system is. Noted authors in the 

field have defined systems as the following: 

"A system is an array of components designed to accomplish 
a particular objective according to plan." [Johnson, 1973] 
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"A system is an organized, unitary whole composed of two 
or more interdependent parts , components , or subsystems 
and delineated by identifiable boundaries from its en- 
vironmental suprasystem. " [Kast, 1974] 

"A system is an organized or complex whole; an assemblage 
or combination of things or parts forming a complex or 
unitary whole." [Cleland, 1975] 



There are about as many definitions of systems as there are 
authors in the area, but all have certain common characteristics. 
The following definition will serve as a composite definition 
of the term "system"; 

"A system is a group of elements, either physical or non- 
physical in nature, that exhibit a set of interrelations 
among themselves and interact together toward one or more 
goals, objectives, or ends." [Alexander, 1974] 

Diagnosis of the above definition of "system" will be use- 
ful and will enable correlation with the management function 
within the Test and Evaluation process. 

The word "elements" may refer to physical objects such as 
component parts of an airplane, or it may also refer to inter- 
related activities such as the managerial functions of planning, 
organizing, directing, and controlling. An "element" of a 
system may also be a lower-order system, hence a subsystem of 
a larger system such as T&E activities within the T&E system. 
Relations that exist among system elements may be examined in 
terras of type of relation present, whether spatial, time, cause 
and effect, physical, logical, and/or mathematical. 

The purpose of a system is shown by its stated goals or 
objectives, the reason why the system was created. When a 
system is actually a subsystem, or a clearly defined integral 
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part of a larger system, its purpose is usually to support 
the major system activity. 

An important concept in systems theory is that of system 
boundaries. Boundaries differentiate between intra- and inter 
systemic relations and only when these boundaries have been 
clearly defined can focus be directed toward system elements 
and interrelations . - 

The system of concern in this thesis is the Test and 
Evaluation system; however some mention of the managerial 
systems within the T&E system will be made. A dynamic model 
will be used to analyze the system and its various characteris 
tics. The dynamic model is one of the many types of models 
which can be used to represent a system; other models are 
summarized in Appendix C. 

B. THE DYNAMIC SYSTEMS MODEL 

Dynamic systems are systems in which certain elements are 
altered in form through an ongoing, self-controlling process 
to create other unique distinct elements [Alexander, 1974] . 
These systems are isomorphic in character and exhibit a pur- 
poseful activity in the achievement of their goals and ob- 
jectives. In general, dynamic systems display the general 
attributes associated with living things; purposeful activity, 
self-regulation, self-direction, and predictable behavior. 
Inputs are transformed into outputs in a given environment. 

A schematic model of the dynamic system as used for this thesi 
is presented in Figure 1. The significant features of the 
dynamic system, as they relate to T&E, are summarized below. 



18 



OBJECTIVE 

DIRECTION 
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DYNAMIC SYSTEMS MODEL 
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1 • System Objectives 



Objectives of a system are needed before the system 
can exist, for they describe the purpose and reason for exist- 
ance of the system, i.e. the direction the system is to take. 
These objectives usually exist in the form of general and 
specific objectives. A general objective may be "to perform 
test and evaluation of the total aircraft" . Specific objectives 
are much more detailed and usually in an exhaustive list. They 
could include objectives such as "conduct test and evaluation 
of aircraft flying qualities, performance, weapons/store 
separation . . . " . 

2 . System Inputs and Outputs 

An underlying characteristic of inputs/outputs is that 
they may be either homogenous or heterogenous in nature. Inputs 
for a dynamic system are defined as one or more elements in a 
given dynamic system that are consumed or transformed during 
the operation of a dynamic process. Outputs are defined as 
one or more elements in a given dynamic system that are created 
by the operation of a dynamic system. Wide disparity in input 
resourses may exist for the T&E system, such as test airplanes, 
funds, and personnel which are used to generate the various 
outputs of reports and experience. 

Dynamic systems usually require a stream of inputs 
which may be either continuous or discrete over a period of 
time in order to function properly. Thus the input/output 
function can be considered as a flow of matter and/or energy 
over a period of time. 
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The relation between the inputs and outputs is of 
importance in describing the dynamic system. This in essence 
means knowing the type and quantity of inputs which are neces- 
sary to produce the required outputs. This relation also 
leads to determination of the system efficiency as will be 
mentioned in a later section. 

3 . System Processor 

The processor is the site or environment where the 
interaction among system elements takes place. It is the 
place where transformation of the inputs to outputs occur. 

The processor may have physical dimensions ranging from a 
very small confined area to a large ill-defined region. It 
could also be conceptual in nature, existing only in the minds 
of men. Processors may be machines, equipments, biological 
units, the human mind, T&E activities. Directorates, or Sections. 
For a T&E activity the processor is composed of elements of 
Directorates, which are systems themselves. They are composed 
of elements of Branches which are composed of elements of 
Sections which are composed of elements of personnel. Each 
of the systems and subsystems are a part of the total T&E 
system process. 

4 . System Controller 

The controller is that element in the dynamic system 
that controls the flow of inputs to the processor and estab- 
lishes the mode of operation for the processor. The controller 
determines the way the system will meet its objectives. He 
can do this by providing guidelines and/or rules by which the 
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system must operate to achieve predictable output of satisfactory 
quality and quantity. Thus the ultimate performance of the 
dynamic system depends to a great extent on the controller's 
design and operation. 

The controller has two separate yet interrelated types 
of activity. He first regulates the flow of inputs in accor- 
dance with predetermined input/output relations to maintain a 
stable, dynamically balanced system. This could include such 
actions as staffing the T&E activity to appropriate levels 
with the proper technically competent people. Secondly the 
controller establishes the processor's overall operating 
characteristics by structuring the procedures and activities 
to be used in the transformation process. This could include 
setting certain quality and quantity standards for work per- 
formance . 

5 . System Control Loop 

The control loop encompasses the entire process from 
input flow to output flow through control by the controller. 
Needless to say, the control of a dynamic system depends on 
the functional quality of the entire control loop. All elements 
of the system must function properly. 

The quality of the control loop performance depends 
on the information derived from systemic outputs. For system 
effectiveness those attributes of the output stream that are 
most important in determining the degree of achievement of 
system objectives or those that define the system operating 
characteristics are the ones that must be measured. 
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The type of control loop involved in T&E systems is 
the "open-loop system" as opposed to the "closed-loop system" . 
The terms "open" and "closed" denote the degree of interaction 
between a given system and its environment. Open-loop systems 
are those in which human beings act as the controller. This 
type of system lacks a substantial degree of self-regulating 
capacity and its overall stability is significantly diminished. 
Thus the managerial capability of the controller, i.e. the 
manager, plays an important function in maintaining system 
stability and effectiveness. 

6 . System Feedback 

The term "feedback" can imply two separate meanings. 
First, feedback may refer to the data gathering process, as 
in gathering information for the decision making process. The 
terra may also denote the direct modification of the input 
stream by the output stream of the system without direct 
controller intervention. The first reference to feedback 
involves the control loop of the dynamic system. Here the 
controller initiates or has control over the feedback process; 
such as questions he may ask or reports he may request. Where 
the controller has no control, as in the latter form of feed- 
back, the process is more mechanical such as the thermostat. 

The T&E systems should consist of the first type of feedback 
discussed, that of information gathering by the controller. 

7 . System Environment 

The system environment is composed of those factors 
that influence the operation of the system, and can be 
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characterized as either stable, dynamic, or turbulent in nature. 
The system cannot afford to ignore the environment if it is 
to survive. 

In the Test and Evaluation world the environment is 
rarely stable, things and/or situations always seem to change. 
Ocassionally the environment is turbulent, as is the case when 
a test airplane is lost. However for the majority of the time 
the environment is dynamic, ever changing, perhaps even in a 
predictable manner. 

It is incvimbent upon the managerial system to recognize 
that external factors to the system do in fact affect the mode 
of operation of the system. Among the most often encountered 
factors are funding changes, additional work load requirements, 

and policy changes by top management. 

1 

8 . System Interface 

System interfaces occur in a compound system environ- 
ment. Compound systems are those which are composed of two 
or more distinct subsystems that operate together through the 
action of a unified controller or manager. Here the processor 
is subdivided into two or more separate units that must function 
together to achieve the objectives of the total dynamic system. 
This is most evident in any organization system such as a 
Directorate which is composed of Branch subsystems, all under 
the control of a Director. 

An underlying characteristic of compound systems is 
the interface required with other systems. Direct interface 
is the point where the output of one system becomes the input 
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of another system. The information interface is another type 

of interface which may exist. In this case the linkage between 

the two systems is informational rather than physical. 

Compound systems may be either series or parallel in 

nature. If the output of one system becomes the input to 

another system, the systems are in series. Such is the case 

of NAVAIRSYSCOM and the field activities. The outputs of 
1 

NAVAIRSYSCOM, the task assignments and funding, become the 
inputs of the field activities. 

If two or more dynamic systems are operating independent 
of each other and their outputs are combined to furnish the 
inputs to another system, the systems are in parallel.. This 
is the case of a matrix organization where the outputs of 
various functional sections are combined to furnish the inputs 
to a Program Manager . 

9 . System Efficiency and Effectiveness 

System efficiency may be simply expressed as performing 
effectively at minimxam expense of resourses. At the Section 
level this entails such things as using different resourse 
saving flight test techniques or data reduction procedures. 
Needless to say, cost reduction and/or cost minimization is 
always of concern in any system. 

System effectiveness is another important parameter. 

As previously stated, every system has a purpose or objective. 
The measure of system effectiveness is directly related to the 
degree to which these objectives of the system are met. Un- 
fortunately in the T&E world this sometimes is a nebulous area. 
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particularily where more than one objective is involved. Such 
is the case when the objectives of a T&E system may not be in 
alignment with the objectives of the contractors or of OPTEVFOR 
during the T&E process of a weapon system. 

Two additional concepts must be considered under systems 
effectiveness. An evaluation criteria must be firmly estab- 
lished before any evaluation of the systems effectiveness can 
be made. Whose objectives are important and in what order? 

The second concept to consider is one 'of ”to satisfice". This 
term may be defined as the satisfaction or fulfillment of needs 
or objectives only up to the point where there is insufficient 
motivation to expend further resourses to achieve greater satis- 
faction. 

Thus the concepts of system efficiency and effectiveness 
are separate and distinct. Together, however, these two mea- 
sures provide a means for evaluation of the total systems per- 
formance. 

C. SYSTEMS CONCEPTS AND MANAGEMENT 

The previous sections have presented the concept of a 
system and the functional characteristics of the dynamic 
systems model. This section will now relate the systems 
approach to the management functions. 

1. Management Defined 

Perhaps everyone has his own definition or conception 
of what management is. Usually it is stated in terms of 
functions that managers do, not what management is. In the 
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context of this thesis management will be defined as "the process 
whereby unrelated resourses are integrated into a total system 
for objective accomplishment" [Johnson, 1964] . Performing the 
function of management is of course done by managers. 

Managers attempt to weld human, technological, and 
capital resourses into an effective, cohesive whole to achieve 
the objectives of the organization. A manager gets things 
done by working with people and physical resourses in order to 
accomplish the objectives of the system. He coordinates and 
integrates the activities and work of others rather than per- 
forming operations himself. 

2 . Management and the Systems Approach 

The effective and efficient function of a system is 
under the auspices of the controller or the manager of the 
system. To effectively perform the function of management 
within the system the manager must be aware of what his system 
is and the role he and it play in the overall "big picture" . 

A good manager relies on ideas, resourses, things, tools, 
and people to achieve objectives. He utilizes such managerial 
functions as planning, organizing, staffing, directing, and 
controlling as prescribed by such authors as Koontz and O'Donnell 
[1968] . In the systems approach to management the managerial 
functions are not performed as separate entities, rather they 
are performed in conjunction with operations of the system as 
a cohesive whole. Any and all operations and functions per- 
formed by managers must be oriented toward the objectives of 
the organization. 
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In effect the systems approach to management provides 
the manager with a framework for thinking about the job of 
management. This approach forces recognition of system ob- 
jectives, inputs and outputs, the environment, interfaces with 
other systems, and overall effectiveness. More importantly 
however it guides the manager to view not only his immediate 
system but the "whole" system. Management and the systems 
approach as related to T&E is addressed in more detail in 
Chapter VI. 

D . SYSTEMS SUMMARY 

A system, no matter how large or complex, must have an ob- 
jective, a purpose for being. It will be composed of inter- 
related elements or subsystems and will consist of an input- 
transformation-output process. The system, in the context of 
this thesis, will function as an open system where exchange of 
information, energy, or material will occur with the environ- 
ment. The system will have boundaries of operation, even though 
at times not explicitely defined. The system will have inter- 
faces, both inter- and intra-system and both functional and 
informational in nature. The system will also have an environ- 
ment in which to function. Thus the above descriptive narra- 
tive is not unlike any organizational system within the T&E 
system process. 

The systems approach provides a systematic way of thinking 
about the job of management and provides a framework for 
visualizing internal and external environmental factors as an 
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integrated whole. It allows recognition of the proper place 
and function of subsystems. It is important to recognize the 
integrated nature of specific systems, including the fact that 
each system has both inputs and outputs and can be viewed as 
a self-contained unit. It is equally important to recognize 
that these systems are a part of a larger system. 
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III. THE TEST AND EVALUATION SYSTEM 



A. GENERAL 

The Test and Evaluation phase for the development of a 
weapon system is composed of many organizational subsystems 
interacting in series and/or in parallel, processing inputs 
into outputs to obtain some objective. Usually this objective 
is the timely deployment of a successful weapon system into 
the Fleet in a cost effective manner. Figure 2 is a much 
simplified, but representative, model of the T&E process. As 
such the model displays all the classic characteristics of the 
dynamic systems model as described earlier. The processing 
function within the T&E system is not however composed of 
actions and interactions with definable points of interface. 
There are role conflicts which tend to stretch out the expen- 
diture of scarce resourses, namely flight test time, and these 
role conflicts eventually work there way down the hierarchy 
of organizational systems and subsystems until they reach the 
point where the scarce resourses are physically managed and 
expended . 

In order to effectively analyze the T&E system and the 
systems approach to management of T&E the concept of T&E must 
first be presented. Material within this chapter is either 
paraphrased or taken directly from DoD Directive 5000.3, CNO 
Instruction 3960.10, the Navy RDT&E Management Guide, and 
the Board of Inspection and Survey Aircraft Test Directives. 
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B. TEST AND EVALUATION DEFINED 

Test and Evaluation is an all encompassing pair of words 

which actually have a rather simple yet all inclusive meaning. 

The following definition of T&E, taken from the RDT&E Management 

Guide will be used throughout this thesis; 

Test and Evaluation is defined as the performance of tasks 
and the evaluation of test results which are conducted 
specifically to: 

1. Support design and development activity. 

2. Measure performance against specified acceptance 
criteria. 

3. Ensure satisfactory operation with related inter- 
facing systems. 

4. Confirm operational effectiveness and suitability. 

5. Validate the adequacy of documentation for support 
and test equipment, personnel training; maintenance 
and operation of the whole system/subsystems and 
other elements . 

The term "evaluation" denotes the review, analysis, and 
report of qualitative and quantitative data produced during 
current or previous inhouse testing, data obtained from 
tests conducted by other government agencies and contrac- 
tors, from operational and commercial experience, or com- 
binations thereof . 



C. BASIC TEST AND EVALUATION POLICY 

Basic T&E policy calls for a development strategy based 
on periodic performance demonstrations. Programs are to be 
structured and resourses allocated to ensure that the demon- 
stration of actual achievement of program objectives is the 
pacing function. Test and Evaluation begins in the earliest 
phase of RDT&E with experimental testing of scientific hypo- 
theses and continues beyond completion of development where 
primary emphasis is on perfecting doctrine for the most effec- 
tive deployment of advanced weapons. 
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A basic T&E concept is that of the "independent evaluation" 
Distinction is made between the Developing Agency (DA), i.e. 
the seller and the operating forces, i.e. the buyer. The DA 
thus does not have unilateral control of the establishment of 
test requirements, the conduct of tests, or evaluation of the 
results. The independent evaluation is performed by OPTEVFOR. 
Members of this force must therefore perform a key role in 
determining test requirements and have access to test results 
obtained by the DA. Evaluation for service use, operational 
effectiveness, and suitability is performed by OPTEVFOR and as 
such have a major impact on the acquisition milestone decisions 
These decisions are made by the Defense System Acquisition 
Review Council (DSARC) . 

The T&E process has recently incorporated the Test and 
Evaluation Master Plan (TEMP) as a controlling management 
document. The TEMP defines the test and evaluation plan for 
each major weapon system acquisition program. It contains the 
integrated test and resour se requirements of the DT&E, OT&E, 
and PAT&E agencies and is updated periodically to reflect re- 
quired changes. 

D. TYPES OF TEST AND EVALUATION 

Department of Defense Directive 5000.3 and CNO Instruction 
3960.10 delineate three basic types of T&E; DT&E, OT&E, and 
PAT&E, each of which is delegated to a different organization. 
Figure 3 will aid in illustratin the relationship and time 
phasing of each system throughout the T&E process. 
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1. Developmental Test and Evaluation 



Developmental Test and Evaluation is planned by, con- 
ducted by or for, monitored by, and reported by the DA. The 
DA shall ensure that the DT&E program and the OT&E requirements 
are integrated into the program schedule and with proper budgeting. 
Also the DA shall provide OPTEVFOR with all significant DT&E 
test data. The objectives of this type of test and evaluation 
are to: 

(1) Demonstrate that the engineering design and 
development process is complete. 

(2) Demonstrate that the system will meet specifi- 
cations . 

(3) Demonstrate that the design risks have been 
minimized. 

(4) Estimate the system's military utility when 
introduced . 



Developmental Test and Evaluation is conducted in four 
major phases. The specific objectives of each phase are deve- 
loped by the DA and published in the TEMP. The phases are: 

(1) DT-I is that DT&E conducted during the conceptual 
phase to support the program initiation decision. 

(2) DT-II is that DT&E conducted during the validation 
phase to support the full scale development deci- 
sion. This phase, normally conducted at the sub- 
system/component level, demonstrates that design 
risks have been identified and minimized. 

(3) DT-III is that DT&E conducted during the full 
scale development phase to support the first major 
production decision. This phase demonstrates that 
the design meets all its specifications. This 
phase may be further subdivided into other phases 
such as contractor and Navy technical evaluations . 
The purpose of the TECHEVAL, the final sub-phase 
is to verify that the design meets specified re- 
quirements and is ready for OPEVAL. 
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C4) DT-IV is that DT&E conducted after the first 
major production decision to verify that the 
improvements or corrections of design defi- 
ciencies discovered during OPEVAL, FOT&E, or 
fleet deployment are effective. 



2 . Operational Test and Evaluation 

Operational Test and Evaluation is accomplished by 
operational and support personnel of the type and qualifica- 
tion of those expected to use and maintain the system when 
deployed, and is conducted in as realistic an operational 
environment as possible, OT&E is planned by, conducted by or 
for, and reported directly to the CNO by OPTEVFOR. Additionally 
OPTEVFOR provides the DA with all significant OT&E test data 
and analysis. The objectives of this type of test and evalua- 
tion are to: 

(1) Estimate the prospective system's military utility, 
operational effectiveness, and operational suit- 
ability (including compatibility, interoperability, 
reliability, maintainability, and logistic and 
training requirements) , and need for any modifica- 
tions . 

(2) Provide information on organization, personnel 
requirements, doctrine and tactics. 

(3) Provide data to support or verify material in 
operating instructions, publications, and handbooks. 



OT&E is subdivided into two major categories: Initial 

OT&E (lOT&E) which is all OT&E prior to the first major pro- 
duction decision, and Follow-on OT&E (FOT&E) which is all OT&E 
after the first major production decision. OT&E is divided into 
five major phases (3 lOT&E and 2 FOT&E) . The specific objectives 
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of each phase of OT&E are developed by OPTEVFOR and published 
in the TEMP. The phases are: 

(1) OT-I is any lOT&E that may be conducted during 
the conceptual phase to support the program 
initiation decision. 

(2) OT-II is that lOT&E conducted during the vali- 
dation phase to support the full-scale develop- 
ment decision. The objectives of OT-II are to: 

(a) Provide an earl-y estimate of projected 
operational effectiveness and operational 
suitability of the system. 

(b) Initiate tactics development. 

(c) Estimate program progress. 

(d) Identify operational issues for OT-III. 

(3) OT-III is that lOT&E conducted during the full 
scale development phase to support the first 
major production decision. OPEVAL is the final 
sub-phase of OT-III. Specific objectives of 
this phase include: 

(a) Demonstration of the achievement of program 
objectives for operational effectiveness 
and suitability. 

(b) Continuing tactics development. 

(4) OT-IV is that FOT&E conducted after the first 
major production decision, but before production 
systems are available for testing. Specific 
objectives of this phase include: 

(a) Testing of fixes to be incorporated in 
production systems. 

(b) Completion of any deferred or incomplete 
lOTSeE. 

(c) Continuing tactics development. 
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(5) OT— V is that FOT&E conducted on production systems 
as soon as they are available. Objectives of 
this phase include: 

(a) Demonstration of the achievement of program 
objectives for production system operational 
effectiveness and suitability. 

(b) OT&E of the system in new environments or 
in new applications or against new threats . 



3 . Production Acceptance Test and Evaluation 

This is test and evaluation of production items to 
demonstrate that the items procurred fulfill the requirements 
and specifications of the procurring contract or agreement. 

This function is carried out by the Board of Inspection and 
Survey (BIS) . BIS is responsible to the SecNav via CNO for 
conducting tests and recommendations for "acceptance for service 
use" if test results warrant such. The purpose of these 
"Service Acceptance Trials" are to: 

(1) Discover and report deficiencies at the earliest 
possible time so that corrective action can be 
taken to ensure that aircraft delivered to the 
fleet will be capable of carrying out the in- 
tended mission. 

(2) Recommend design changes which could improve system 
effectiveness and identify design features which 
should be changed or avoided in future designs. 

(3) Determine if contract specifications and authorized 
changes thereto have been satisfactorily fulfilled 
and render an opinion as the the government's or 
the contractors responsibility for the correction 
of defects. 

(4) Recommend to the SecNav the conditions of accep- 
tance or rejection of an aircraft for service 
use. 
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E. TEST AND EVALUATION SUMMARY 



The T&E process is thus presented as a hierarchy of systems 
all acting within the overall T&E system process. The overall 
T&E system is composed basically of three large systems, the 
DT&E, OT&E, and PAT&E systems. Each of these systems have ob- 
jectives and definitive time phases within the T&E process. 

Thus the WHO , WHAT, WHY , and IVHEN of the T&E process is provided. 
A typical organization chart of the three systems involved in 
a weapon system program is presented in Figure 4, and it displays 
an important concept to recognize in applying the systems ap- 
proach to management. That is as the T&E system becomes larger 
and more organizations are , involved in T&E due to their special- 
ization the number of systems interface and combinations thereof 
increase enormously. 
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FIGURE 4 

ORGANIZATION CHART OF THE SYSTEMS INVOLVED IN 
TEST AND EVALUATION OF A WEAPON SYSTEM 
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IV. A SYSTEMS LOOK AT THE TYPES OF TEST AND EVALUATION 



A. GENERAL 

Modern T&E has come a long way from the scarf and goggle 
days of screaming power dives and high-g pull outs. Technology 
in data acquisition systems and analysis procedures have kept 
up with the ever increasing weapon systems technology and 
complexity. The T&E process has grown from one lieutenant per- 
forming all flight tests on a system to several different and 
distinct organizations performing flight tests on a weapon 
system such as the F-14 or F-18 airplanes. Growth and sophis- 
tication of the T&E process has not however been without problems. 
Committees and directives' have provided guidelines and direction, 
but this growth has often created significant problems; problems 
which ultimately mean inefficient and ineffective use of T&E 
scarce resourses. The following sections attempt to reconstruct 
some of the more recent and significant adverse factors affecting 
the overall T&E process. 

Recent theses by H. L. Young, Jr. [NRG Thesis, 1973] and 
W. C. Bowes [NPG Thesis, 1974] studied many aspects of the 
overall T&E process as relating to F-14 managerial problems and 
to the organization of T&E in general. Some of their findings 
and conclusions are included in the following sections to add 
more substantiation to the managerial factors affecting T&E. 

The following sections are organized into the types of 
T&E performed on a weapon system; DT&E, OT&E, and PAT&E. Each 
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of these systems are subsequently divided into subsection of 
objectives, inputs/outputs , interfaces, system environment, 
and effectiveness, significant elements which comprise the 
systems approach. A later chapter will look at the three 
systems together in the total T&E process system and the 
associated objectives and interfaces. 

B. DEVELOPMENTAL TEST AND EVALUATION 

The functions and operation of DT&E have been in existence 
since the first tests were conducted on a Navy airplane. Their 
objectives, interfaces, and environment have evolved through 
time and can be considered as "lodged in concrete" in many 
instances. Figure' 5 provides a schematic model which will aid 
in the following "systems analysis" of DT&E. The four small 
circles represent systems which are elements of the larger 
DT&E system. The DT&E system is existing in the environmental 
conditions as depicted on the figure. The arrows of the indi- 
vidual systems are intended to signify only relative direction 
of objectives with respect to each other and not magnitude, 
rate, or velocity. 

1 . System Objectives 

The objectives of a DT&E organization are usually well 
defined and parallel those listed under paragraph III-D-1. Each 
T&E activity and Directorate within the activity has mission 
statements and/or objectives of the organization listed. Addi- 
tionally, work units from NAVAIRSYSCOM or other higher organi- 
zations supply specific direction and objectives for specific 
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projects. Finally, test objectives of the weapon systems 
exist in the form of specifications and performance guarantees. 

Note that Figure 5 implies that the elemental sub- 
systems comprising the DT&E system may have objectives in 
varying directions with respect to each other. Herein lies 
the major problem that the T&E process must face and will be 
treated in the next chapter. 

2 . System Inputs/Outputs 

The inputs of the DT&E organization are well defined 
in terms of funds, personnel, and test resourses such as test 
airplanes or airplane subsystems . However the inputs may or 
may not be consistent with the overall objectives of the program 
in concern. For instance, funds may not be adequate or test 
airplanes may not be properly configured to make meaningful 
tests . 

Outputs of the DT&E system are usually reports of test 
results. Timeliness and content of reports have been reported 
as areas of significant concern. Young [Thesis, 1973] and 
Bowes [Thesis, 1974] reported that project managers felt that 
T&E reports are often submitted too late for timely program 
decisions. Additionally, data submitted during developmental 
phases, such as the contractor bi-weekly or bi-monthly reports 
were submitted six to eight weeks after being generated, often 
too late for effective utilization. 

The problem of un-timely data occurs for several reasons. 
One reason is the scope and complexity of required tests and 
data acquisition ■ and handling problems. A second is the lack 
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of feedback and dialogue. Another is the existence of the 
review chain, a necessary evil in the promulgation of a T&E 
report. It is the authors view however that the major reason 
may be because of the lack of understanding or acknowledgement 
of the role that the DT&E activities play in the overall T&E 
process. Each output report becomes a cog in a wheel somewhere 
higher up the T&E organizational hierarchy. It is incumbent 
on the management of the lower systems to realize that they play 
a significant part (however small) in the overall process and 
their system outputs must be timely to be effective. 

3 . System Interfaces 

System interfaces occur where there is direct inter- 
action between two or more different systems or where there is 
an information flow from one system to another. As expected, 
the T&E process is composed of many interfaces. There are 
interfaces between DT&E, OT&E, and PAT&E. And within DT&E there 
are interfaces between the T&E activities, the contractors, and 
NAVAIRSYSCOM. For instance, T&E of the F-14 was composed of 
elements of activities such as NAVAIRTESTCEN , PACMISTESTCEN , 
and the contractors. Two major problems exist in this area of 
concern; role conflicts between T&E systems and system parochia- 
lism. 

Role conflicts exist because of inadequately established 
objectives or mission statements or because testing of a weapon 
system is done by two or more different activities separated 
by large geographic distances. Role conflicts arise from the 
historical evolution of the various activities. Each activity 
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in time builds up an expertise based on what their overall 
mission is and what they perceive is required to accomplish 
that mission. Thus conflicts arise on who tests what, and the 
results is usually redundant testing. 

In the weapon system area of the F-14 program this 
conflict was addressed and a Joint Evaluation Team (JET) was 
formed consisting of members from BIS, NAVAIRTESTCEN , PACMIS- 
TESTCEN , and from the OT&E community. The JET proved successful 
to some extent (only within the DT&E community) but conflicts 
within DT&E activities still existed. The root of the evil 
could be attributed to the syndrome known as parochialism. 

As reported by Young [Thesis, 1973] the self-serving 
parochial attitude sometimes motivated the actions of organi- 
zations participating in the F-14 T&E. This syndrome is diffi- 
cult to dociament and validate, and some feel that a certain 
amount of parochialism is good, that it creates a competitive 
spirit among T&E systems. But the important point to remember 
about parochialism is that it seems to breed mutual distrust 
between organizations, thus curtailing efficient and effective 
pursuit of overall objectives. Role conflicts and parochial 
attitudes are caused to some extent by a lack of understanding 

of "who you are" , what function do you play in the overall T&E 

/ 

process; and a general lack of concern for the primary objectives. 

Interfaces must exist for the T&E process, but boundaries 
of each system must be recognized and respected to minimize 
role conflicts and parochialism before the interfaces can be 
effective. This means that the individual system objectives 
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must be established as to purpose and scope such that a inter- 
related systematic network of systems can be established with 
appropriate interfaces . 

Interfaces are facilitated by the well known "list of 
key personnel", but this list does not itself establish success- 
ful interaction and interface. A system must first recognize 
that interface with other systems is necessary to meet the ob- 
jectives of that system and of the objectives of the total 
system. This system must then establish interface with others, 
or a linking within the T&E system. 

4 . System Environment 

A primary function of the system is to recognize that 
the system exists within an environment. This environment 
changes, either dynamically or turbulently in nature and the 
system must react and adjust to survive. For the DT&E system 
the environment consists of such things as top management policy, 
available funds, directives, OT&E requirements, and PAT&E re- 
quirements to mention only a few. The manager of the system 
guides the system and controls its destiny, but the environment 
directs and controls the manager. 

5 . System Effectiveness 

As previously stated, system effectiveness is a measure 
of how well the objectives of the system have been met. Thus 
an effective DT&E system is one which performs its mission and 
meets its objectives. More importantly however the author sug- 
gests that an effective DT&E system is one which meets its 
objectives in the context of the total T&E process. The system 
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views its function as a part of an interrelated group of T&E 
systems performing a function to meet the overall objectives 
of the T&E process of the weapon system in concern. An import- 
ant factor missing in this context is an evaluation criteria. 

How well must a system perform or meet its objectives to be 
effective? 

A similiar situation exists when a T&E system is testing 
a weapon system. Criteria for measuring weapon system effective- 
ness are supplied in the form of specifications and guarantees. 
Note however that the fact that a weapon system may meet all 
the specifications does not guarantee an effective system. This 
is not a problem of objectives not being stated; but is a 
problem of the objectives being erronously stated or misinter- 
puted. 

It is well recognized that system effectiveness in the 
T&E world, whether involving a management or a weapons system, 
is not easy to evaluate. A weapon system has the potential of 
being numerically evaluated thus providing some measure of 
effectiveness. The author feels however that the evaluation 
of a management system is a subjective factor, one that requires 
rationalization by the manager and a view of the whole instead 
of a view of individual parts. 

C. OPERATIONAL TEST AND EVALUATION 

The functions and operations of contemporary OT&E have just 
recently evolved (since 1971) and hence are short on tradition 
but long on directive policies. The OT&E primary function of 
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"independent evaluation" has been both praised and criticized 
hence their place in the overall T&E process is worthy of 
analysis. Figure 6 will aid in the following "systems analysis" 
of OT&E. 

1 . System Objectives 

The objectives of the OT&E system are, as in DT&E, well 
defined and parallel those listed under paragraph III-D-2. Each 
activity within OT&E, such as AIRTEVROR FOUR (VX-4) , have 
specific objectives such as to perform OT&E on fighter aircraft 
systems. Another squadron, VX-5, performs OT&E on attack air- 
craft systems. However, one of the deficiencies within the 
overall T&E process as reported by Bowes [Thesis, 1974] is 
the "lack of adequate OT&E test objectives". This problem 
still exists and is further amplified because of the lack of 
adequate criteria for which to design test scenarios; speci- 
fications and guarantees do not exist for combat profiles or 
for operations conducted in war zones . 

2 . System Inputs/Outputs 

The OT&E community has much the same problems as the 
DT&E community with respect to funds, personnel, airplanes, 
and reports. Peculiar problem areas to OT&E however lie in 
the area of airplanes and personnel. 

Being operationally oriented necessitates the use of 
production configured airplanes. This unfortunately is not 
always possible in the T&E process until far down the develop- 
ment cycle. Hence a true operational assessment of a weapon 
system cannot be made until dangerously close to the milestone 
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for major production decision. The role and objectives of OT&E 
are thus difficult to achieve in a strict sense. Of necessity, 
flying has to be performed on prototype airplanes with systems 
possibly still in development. The final production configured 
airplane may not be the same as tested by OT&E. 

Lack of analytical and technical personnel is another 
factor adversly affecting the function of OT&E. Program planning 
and data analysis support is currently being performed by 
non-airframe type contractors working under the control of 
OPTEVFOR. This is expensive and undersirable in some respects. 
These functions could just as well be performed by civil service 
engineers with the appropriate expertise. 

3 . System Interfaces 

Interfaces within the OT&E system are primarily with 
the squadrons, Headquarters, DT&E activities, and the contractors 
who perform data analysis. An apparent interface problem, as 
perceived by this author, may lie within the interface of 
Headquarters and the squadron level. This is a result of 
possible differences of opinion as to the interpretation of 
the stated objectives. An educational and/or communicational 
in-house process can rectify the situation. 

A unique interface problem facing OT&E is the dilemma 
of when to get involved in the T&E of the Weapon system. Too 
early an entry may mean disruption of the developmental process 
and may involve a prototype airplane non- representative of the 
production lot. Too late an entry and OT&E will have little 
or no input on the major production decision and suitability 
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fleet operation. The author perceives internal conflict 
within DoD on the exact role of OT&E and further still, the 
author perceives conflict within the OT&E conununity itself on 
what it's role is to be. 

4 . System Environment 

The environment that OT&E operates within is much like 
that of DT&E. The major factors being CNO policy/directives, 
DT&E, PAT&E, funds, and test aircraft availability. The OT&E 
system too must recognize that the environment exists and is 
ever changing in order to survive. 

5 . System Effectiveness 

As with the DT&E system, the effectiveness of the OT&E 
system is a function of how well the system meets its objectives 
and how well it fits into the interrelated group of other T&E 
elements. Managerial effectiveness can be attained by recogni- 
zing this, just as with any other T&E activity. 

The major problem lies in the lack of test objectives 
and criteria for determing the effectiveness of the weapon 
systems themselves when evaluating in an operational environ- 
ment. A test program should be designed to measure the suit- 
ability or effectiveness of a system against some criteria 
which may either be supplied by an Operational Requirement or 
Detailed Specification. An operational effectiveness criteria 
is needed for which the OT&E system can evaluate the results 
of their tests. 

Thus the objectives of OT&E are stated, but the missing 
ingredient is a test criteria. The criteria of "mission 
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suitability" is well and good but different pilots have differ- 
ent views as to what is suitable, and more importantly, different 
abilities to evaluate the system. By the "seat of the pants" 
is not good enough, hence the effectiveness of the weapon 
system is at best a subjective . evaluation. This problem is 
however not unique to OT&E. The DT&E community is hard pressed 
to evaluate system effectiveness as previously discussed. Meeting 
or exceeding a specification number does not make the system 
effective. Apparently the weapons planners do not know enough 
yet about threat forecasting and combat effectiveness criteria 
to tie a number to it, and under standly so; the enviroiment 
for which the system is designed changes constantly. 

D. PRODUCTION ACCEPTANCE TEST AND EVALUATION 

The functions and objectives of PAT&E have evolved along 
with DT&E throughout the years and are carried out by the BIS. 
Figure 7 will aid in the analysis of the PAT&E system. 

1 . System Objectives 

The objectives of PAT&E as carried out by BIS are those 
listed under paragraph III-D-3 and need not be reiterated here. 
Primarily the role of BIS is that of recommending conditions 
of acceptance or rejection for service use of a weapon system. 

This is done by providing timely inputs of technical aspects 
of the weapon system to SecNav via CNO. These objectives have 
been with the total T&E process for some time and tend to assure 
an overall satisfactory weapon system for the fleet. Additionally, 
test objectives exist in the form of specifications and guaran- 
tees for which the weapon system must meet. 
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FIGURE 7 

MODEL OF THE PRODUCTION ACCEPTANCE 
TEST AND EVALUATION SYSTEM 
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2 . System Inputs/Outputs 

Inputs for the PAT&E system include airplanes, funds, 
and T&E activities. The BIS tasks appropriate T&E activities 
to determine compliance with weapon system requirements. Per- 
sonnel from these T&E activities in effect become a member of 
the "BIS team" and actually work for the Board. Funds and air- 
craft are requested from NAVAIRSYSCOM by BIS who then gains 
control of these assets. 

The outputs from BIS consists primarily of individual 
"Yellow-Sheets" which describe specification and/or mission 
deficiencies of the weapon system and must be supported by 
substantiating data. These "Yellow-Sheets" are followed up 
with a formal all inclusive report v/hich is endorsed through 
the hierarchy to the SecNav. 

3 . System Interfaces 

Interfaces within the PAT&E system are much the same 
as in the DT&E system with the exception that the "Board" is 
a primary elemental organization. Note that BIS itself is not 
a testing agency or testing complex per se, it is a reporting 
agency. The existing T&E activities and facilities do the work 
for BIS as BIS members and report test results in required 
form to the President of the Board who in turn reports to SecNav 
via the CNO. 

4 . System Environment 

The environment of PAT&E is much like that of DT&E and 
OT&E. The primary factors being CNO policy, funds, test air- 
planes, and the individual T&E activities. 
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5. System Effectiveness 



Effectiveness of the PAT&E system can be evaluated in 
terms of how well it meets its objectives previously discussed. 
The problem is that no one knows the deficiencies and/or prob- 
lems that are not discovered, only those that are discovered. 
Hence the effectiveness of the PAT&E system may be a measure 
ultimately of successful fleet operation. If the weapon system 
functions well, then the entire T&E process has been performed 
well. If deficiencies are discovered by the fleet during 
operations that something faultered during the T&E cycle. Of 
course these are again subjective areas of discussion and again 
the point of evaluation effectiveness of a T&E system is a 
difficult one to attempt. 

E. SUMMARY OF THE TYPES OF TEST AND EVALUATION 

The preceeding sections have presented a systems view of 
the three types of T&E; DT&E, OT&E, and PAT&E, each of which 
have established objectives. However these three systems are 
comprised themselves of many lower order systems which in many 
cases may have objectives not in congruence with the larger 
systems objectives. The salient point is however that each 
system can be analyzed in terms of typical systematic charac- 
terisitcs of the dynamic systems model, i.e. objectives, inputs 
and outputs, interfaces, environment, and effectiveness. 
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V. A SYSTEMS LOOK AT THE TOTAL T&E PROCESS 



The three types of T&E have been analyzed using a systems 
approach. Each individual system has its fundamental elements 
and systematic characteristics. But the T&E process must be 
viewed as a whole instead of as individual parts. The systems 
of DT&E, OT&E, and PAT&E are all interrelated elemental systems 
of the whole T&E system process. 

The T&E process within the Navy is currently in a flexing 
posture. New policies and trends coupled with stricter control 
over scarce resourses triggered by the economical and political 
environments are impacting T&E. significantly. Rapidly advancing 
technology in both the weapon systems themselves and the media 
and methods for conducting T&E are adding to the complexities 
of T&E today. A systems look at the total T&E process as it 
exists now is warranted, for it illustrates several significant 
problems and areas of conflict. The recent experience on the 
F-14 weapon system and some of the experience to date of the 
F-18 program offer evidence that some systematic approach is 
indeed needed. Figure 8 is presented to depict the current 
environment and turmoil within the T&E system and to aid in 
understanding the succeeding paragraphs. The objectives, 
interfaces, and effectiveness features of a dynamic system are 
again used to analyze the total T&E process. The input/output 
and environmental dimensions are not presented here because they 
expand the systems study to the Congressional level and impose 
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FIGURE 8 

MODEL OF THE TOTAL TEST AND EVALUATION SYSTEM 
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correspondingly immeasurables upon the study such as bureau- 
cratic politics. This expansion, although certainly valid, is 
not needed to put forth this thesis. 

A. SYSTEM OBJECTIVES OF T&E 

The objectives of DT&E, OT&E,and PAT&E have been listed 
and discussed in the preceeding chapter. This section presents 
the objectives of the total T&E process. In simplistic and 
condensed form, the primary objective of T&E is to perform 
tasks and evaluate test results which are conducted to: 

(1) Support design and development activities. 

(2) Measure performance against specified acceptance 
criteria . 

(3) Confirm operational effectiveness and suitability. 

(4) Ensure satisfactory operation with related inter- 
facing systems. 

(5) Validate the adequacy of documentation for support 
and test equipment, personnel training; maintenance 
and operation of the whole system/subsystem and other 
elements . 

Note that objective (1) corresponds to DT&E functions; 
objective (2) to DT&E and PAT&E; and objective (3) to OT&E 
and PAT&E. Objectives (4) and (5) are unfortunately sometimes 
not addressed, either because of ignorance or of constraints 
on the total flight test program itself such as lack of a fully 
configured aircraft. Thus the overall objectives of T&E are 
separated into the three types of T&E systems. The subsequent 
integration of these objectives toward the whole system objecr 
tives are sometimes overlooked. 
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B. SYSTEM INTERFACES WITHIN T&E 

The elemental systems comprising the T&E system which 
interface with each other consist of the DT&E, OT&E, and PAT&E 
systems as presented in the previous chapter. As evidenced in 
Figure 8 these systems are, unfortunately, not always inter- 
related into a cohesive whole necessary for effective T&E. Role 
conflicts exist between the systems and some guidance, direction 
and/or framework for thinking must be implanted in the T&E 
process to minimize this inefficient use of scarce resourses. 

Several areas of conflict within the F-14 T&E program were 
documented by Young [Thesis, 1973] and are summarised here. 

One area of conflict is between the DT&E and OT&E systems. The 
crux of the problem lies within the question of when should 
OT&E become involved in the overall T&E process, if at all. 

One argument says that little OT&E is needed since the DT&E 
community is able to conduct tests with an objective, inde- 
pendent approach by operationally experienced flight crews, 
just as OT&E does. There are fears that OT&E can disrupt the 
DT&E process with too early an involvement and also be counter- 
productive to the OT&E objectives because the airplane systems 
configurations are usually not stabilized until far downstream 
in the T&E process. 

The OT&E community feel, of course, that their involvement 
in the T&E process is necessary, for that is their purpose in 
being. The primary concern of OT&E is apparently not one of 
early involvement, but one of adequate and timely involvement 
prior to the first major production decision milestone. Too 
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meet their objectives the OT&E phase needs calender time within 
the DT&E process and dedicated test airplanes configured as 
close as possible to production configurations. 

Another area of conflict is between PAT&E and OT&E. Both 
systems report to CNO on the adequacy of the total weapon 
system. The OT&E system reports on "adequacy of service 
suitability" while PAT&E (BIS) reports on "acceptance for 
service use" . Even though the OT&E system reports on opera- 
tional effectiveness and PAT&E reports on technical effective- 
ness both realistically offer the objective opinion of whether 
the system is ready for the fleet. However the OT&E system 
reports directly to CNO throughout the T&E process while PAT&E 
does not report until further downstream the T&E process. The 
OT&E system is thus visible to CNO somewhat before the PAT&E 
system. The F-18 program currently has BIS scheduled after the 
first major production decision milestone, certainly contra- 
dictory to the objectives of PAT&E. The BIS recognizes that 
the PAT&E system has little effect on the major production 
decision process if the T&E system exists as described. Hence 
BIS needs an "initial" look at the weapon system in order to 
function consistently within their stated objectives. This 
"Initial Trials- Phase" has long been in existence and should 
remain actively so such as to allow BIS timely meeting of its 
objectives . 

C. SYSTEM EFFECTIVENESS 

As previously discussed, effectiveness is a subjective 
parameter in the T&E world. The individual T&E subsystems. 
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(DT&E, OT&E, and PAT&E) strive to meet the stated objectives of 
their respective systems and a subjective statement as to the 
adequacy of meeting those objectives can be made. Each system 
acts within its own environment and functions accordingly. 

The effectiveness of the total T&E system is just as 
illusive as that of the individual systems comprising T&E. 

The effectiveness of the T&E system as a cohesive whole is 
in question. Evidence to date indicates that the T&E system 
is low in effectiveness because of existing conflicts which 
tend to detract from the overall goals and waste significant 
amounts of scarce resourses. The F-14 JET concept was an 
example. The JET was formed to integrate all testing require- 
ments. All three T&E systems were involved but only the inter- 
face of DT&E/PAT&E was fluid (and only to a minimal degree) . 

The objectives of the JET, as viewed by DT&E, were to satisfy 
the BIS requirements. The means to the end were however in^ 
terpretated differently by the various individuals within the 
DT&E system. The OT&E system, even though involved "on paper" 
within the process, was not readily accepted. Hence their 
objectives fell by the wayside. Thus the objectives of the 
JET concept, to unite the T&E systems, failed. 

There are some indications that the future F-18 program is 
encountering conflicts within the total T&E system. Of signi- 
ficance is the situation within DT&E and the T&E activity sub- 
systems where the missile expertise of the weapon system are 
not currently actively participating within the DT&E process. 
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Conflict is good, too a degree. Without conflict certain 
points of view and alternatives would not be fostered. However 
the result of conflict within the T&E process is rapid, needless 
expenditure of the most significant resourse for T&E, namely 
flight test time. Conflict must be controlled so that the 
total objectives of the T&E system are visible, and each of 
the subsystems within the T&E system align its respective ob- 
jectives to the primay objective of the total system. The 
T&E system must be composed of interrelated parts acting as a 
cohesive whole following a given logical sequence. The three 
systems are interdependent on each other for effective overall 
T&E of a weapon system. 

D. SUMMARY OF THE TOTAL T&E PROCESS 

The total T&E system process is comprised of the DT&E, OT&E, 
and PAT&E systems. Even though the systems have complementary 
stated objectives (in directive form) their actual behavior 
is such that one might conclude that the objectives of these 
systems are not really complementary and do not act in total 
alignment with the objectives of the T&E process. Role conflict 
and parochialism exist and each system must recognize its part 
in relation to the whole before an effective T&E system can 
exist. 
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VI. THE SYSTEMS APPROACH TO MANAGEMENT AND THE T&E PROCESS 



Thus far this thesis has established that the T&E process 
is in fact composed of three large systems interacting to per- 
form the test and evaluation function. These larger systems, 
the DT&E, the OT&E, and PAT&E systems are themselves composed 
of many lower order systems. It has been shown that all these 
systems, including the T&E system itself, displays the systemic 
characteristics of a dynamic system. The concept of the systems 
approach to management has been put forth and the importance of 
viewing the system as a whole has been stressed. As previously 
pointed out, before the manager of a T&E system can effectively 
manage he must know the process of his system and its role in 
the overall T&E system. 

A. MANAGEMENT AND THE TEST AND EVALUATION PROCESS 

The management functions within the T&E process can be 
better understood if the T&E system is further analyzed. 

Chapters III through IV have discussed the types of T&E from 
a systems view. T&E can also be analyzed by looking at its 
various phases. As outlined by Perry [1976] in discussing the 
role of T&E, the T&E process consists of five basic phases: 

(1) Planning 

(2) Programming 

(3) Budgeting 

(4) Testing 

(5) Evaluation 
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A good manager relies on ideas, resourses, things, tools, 
and people to achieve objectives. He utilizes such managerial 
functions as planning, organizing, staffing, directing, and 
controlling as prescribed by such authors as Koontz and 
O'Donnell [1968], More importantly these functions must be 
harmonized and put in juxtaposition with the five basic phases 
of T&E. This correlation is depicted in Figure 9 which shows 
the relationship of basic managerial functions to the basic 
phases of the T&E process. Perry describes the basic phases 
of T&E in the following manner. 

During the planning phase the role of test and evaluation 
is to work for, work with, and provide feedback to the planning 
arm of the organization. As such, seven steps are required 
of the manager for the planning function: 

(1) Understand the need of the program manager. 

(2) Set test and evaluation objectives in harmony with 
the program objectives. 

(3) Develop time phasing for entire life cycle of the 
program. 

(4) Establish test program goals and milestones. 

(5) Select appropriate procedures and methods to be 
applied during T&E. 

(6) Construct a test and evaluation master plan which is 
documented transmittable to other organizations as 

a TEMP . 

(7) Share with other managers in formulating the strategies, 
tactics and policy to achieve success of the program. 

The programming phase is concerned with a quantified defi- 
nition of what work has to be done and what work resourses 
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T&E ACTIVITIES 

1. PLANNING 

UNDERSTAND NEED 
SET OBJECTIVES 
TIME PHASING 
GOALS AND MILESTONES 
SELECT METHODS 
CONSTRUCT TEMP 
DEVELOPPOLICY 



2. PROGRAMMING 

ESTABLISH T&E STRUCTURE 
DELINEATE WORK RELATIONS 
PRICE WORK RESOURCES 
CHOOSE BEST PROGRAM 



3. BUDGETING 
ALLOCATE QUALITY 
DETERMINE QUANTITY 
APPORTION TIME 
BUDGET MONEY 

4. TESTING 

FORMULATE CRITERIA 
DESIGN THE EXPERIMENT 
FIRM THE TEST MODEL 
CONDUCT TESTS 
ANALYZE DATA 



5. EVALUATION 

SYNTHESIZE DATA 
COMPARE TO CRITERIA 
DERIVE RESULTS 
REPORT 
SHARE IN RISK 



FIGURE 9 

RELATIONSHIP OF MANAGERIAL 
FUNCTIONS TO THE PHASES OF TEST AND EVALUATION 
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t>e necessary to accomplish a viable test and evaluation 
program. The basic managerial functions of organizing and 
staffing are important for this phase; as such there are four 
basic steps: 

(1) Establish the test and evaluation work structure by 
making sure that all the system elements (both verti- 
cal and horizontal) are properly put in perspective 
as to what tests and what evaluations must be per- 
formed. 

(2) Delineate the work relationship by identifying how 
the human element and the organizations (internally 
and externally) will perform the structured work. 

(3) Price the work resourses by estimating the costs and 
benefits to be provided by test and evaluation. 

(4) Choose the best program alternative and provide a 
feedback to the program manager for his acceptance. 



The budgeting phase follows planning and programming. The 
test and evaluation organization must be concerned with the 
quality level of the product, the quantity of T&E information 
to be generated, the quantity of resourses to be applied in 
reaching the desired level of confidence regarding critical 
unknowns and these into explicit schedules. This permits the 
monetary and financial budgeting requirements for the total 
T&E program. As such, four basic steps concerned with alloca- 
tion of the budget are: 

(1) Quality 

(2) Quantity 

(3) Time 

(4) Money 
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The pattern of the testing and the evaluation functions 
are derived from the scientific approach of management. During 
the testing phase five activities are required: 

(1) Formulate the evaluation criteria. 

(2) Design the experiments. 

(3) Identify exactly how elements of hardware, software, 
and "brainware" will be integrated during the test 
program. 

(4) Conduct the tests. 

(5) Analyze and process the required data. 

The evaluation phase continues the testing process by 
accomplishing five more steps: 

(1) Synthesize the data obtained from testing by making 
sure that the information is relevant, germaine, 
and valid. 

(2) Compare the synthesized data with performance stan- 
dards and the evaluation criteria. 

(3) Derive the results required by management. 

(4) Report the results in language useable and under- 
standable by management. 

(5) Provide recommendations as to confidence and overall 
risk of the system or product. 

Collectively this is a closed loop process which relies on 
ideas, resourses, things, tools, and people. Such a test and 
evaluation process model can serve the T&E manager as a check- 
list of systematic and sequential steps within his organization. 
In the T&E system the manager must recognize that the objectives 
to meet through use of resourses, ideas, and people are the 
ones consistent with the "whole" T&E process, which itself 
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consists of the basic phases of planning, programming, bud- 
geting, test, and evaluation. The manager must also not 
loose site of the fact that the T&E system is merely a sub- 
system itself. T&E is a managerial tool to be used to provide 
information necessary to make development, procurement, and 
deployment decisions. 

B. STRATEGY FORMULATION - IMPLEMENTATION OF THE SYSTEMS APPROACH 

The task of how to implement the systems approach to manage- 
ment becomes an important issue, for merely talking about the 
concepts and methodology will not create action. This approach, 
once its validity and usefulness have been recognized, must be 
implemented. Implementation can be made to any system for each 
system is independent as to "how" it is managed. Systems are 
not independent as to "how" they function, for each has inter- 
related elements forming a whole. A lesson from the corporation 
world on the concepts of strategy is worthy of note. 

The systems approach to management may be initiated by 
applying the concepts of strategy formulation practiced by 
many corporations. Corporate strategy can best be defined, 
according to Uyterhoeven [1973] by looking at the purposes 
it serves; to provide both direction and cohesion to the enter- 
prise. Providing direction is the traditional objective of 
corporate strategy: to give the company a sense of purpose 

and mission. Providing cohesion is however more essential than 
providing direction. In providing cohesion, corporate strategy 
influences the unit strategies rather than permitting the 
latter to shape the corporate response. Furthermore, to 
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establish cohesion corporate strategy must establish priorities 
among the units. Finally cohesion requires that the activities 
of the various units be interrelated. This integration aspect 
not only provides a common sense of direction to the units but 
also permits the total organizational response to be more ef- 
fective than that of the sum of its parts. These concepts 
from Uyterhoeven about corporate strategy are readily applicable 
to any T&E organizational system. The model developed by 
Uyterhoeven to formulate strategy is presented below with 
amplifying statements shov/ing the relationship to the T&E 
process. The formulation of corporate strategy encompasses 
seven steps: 

(1) Develop a Strategic Profile which concerns itself 
with the business or objectives of the system. 

(2) Evaluate the Environmental Dimension , including 
such factors as politics, economics , and technology. 

(3) Arrive at a Strategic Forecast which relys on the 
validity of the environmental analysis. This concerns 
itself with predicting the future shape or posture 

of the environmental dimension. 

(4) Perform a Resourse Audit which focuses on the internal 
dimensions. Resourses are identified and evaluated 
and the strengths and weaknesses of the organization 
are developed. 

(5) Develop Strategic Alternatives which concerns itself 
with how can the organization respond to a given 
stimuli, what are the range of choices for meeting 
an obligation or objective. 

(6) Perform a Test of Consistency which relates what the 
system or organization is "able" to do with respect 
to its available resourses to what is "possible" in 
its given external environment. 

(7) The manager makes a Strategic Choice of the available 
alternatives which provide consistency within the 
stated goals or objectives. 
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The author must however caution the readers not to over- 
simplify the issues just discussed. Implementation of a 
"corporate strategy" which in turn initiates the application 
of the systems approach to management may meet unsurmountable 
obstacles. These implementation problems are discussed by 
Allison [1971] in his analysis of the "Cuban missile crisis". 

In essence, the rational alternatives and choices arrived at 
may be strongly affected by organizational policies and bureau- 
cratic politics, either adversely or reinforcingly , depending 
on the situation and the people involved. 

C . SUMMARY 

The systems approach provides the manager with a framework 
for thinking about the job of management. Coupling this approach 
with the systematic nature of the T&E process offers an ideal 
tool for the manager to use. The classic functions of manage- 
ment can be kept in juxtaposition with the phases of T&E to 
provide a better insight to management v;ithin the T&E process. 
Implementation of a strategy is perhaps the most important 
first step in practically using the systems approach and as 
such stimulates the manager to think in the proper perspective. 
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VII . CONCLUSIONS 



This thesis has developed two basic conclusions as a result 
of utilizing a dynamic systems model to analyze the Test and 
Evaluation process. The first is that the Test and Evaluation 
process can be represented as a conglomerate of lower-order 
systems. Each layer of systems can be characteristically 
analyzed by focusing on the objectives, input/outputs, inter- 
faces, environmental, and effectiveness dimensions. 

The systematic characteristics of the Test and Evaluation 
process suggests that a systematic approach to management can 
be readily applied by managers. This thus constitutes the 
second basic conclusion of this thesis. A paradigm for the 
Test and Evaluation process, useful in many respects, is the 
systems approach to management. This approach provides the 
managers a systematic approach to the job of management, one 
which the author finds appealing to the engineering background. 
This framework of thinking guides the manager to view the 
whole instead of only his elemental part. More importantly 
the manager will recognize the importance of properly aligned 
objectives. The formulation of a strategy becomes important 
in implementation of this paradigm for it provides the means 
to initiate this approach to management and it provides the 
groundwork for direction and cohesion so needed by the system. 
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APPENDIX A 



ORGANIZATIONS WITH WHICH INTERVIEWS VIERE CONDUCTED 

Developmental Test and Evaluation 

1. Naval Air Test Center, Patuxent River, Md. 

2. Pacific Missile Test Center, Pt Mugu, Calif. 

3. Air Force Flight Test Center, Edwards AFB, Calif. 

4. Naval Air Systems Command, AIR 5102 Office, Wash. D.C. 

Operational Test and Evaluation 

5. AIRTEVRON FOUR (VX-4) , Pt Mugu, Calif. 

Production Acceptance Test and Evaluation 

6. Sub Board of Inspection and Survey, Patuxent River, Md. 
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APPENDIX B 



EVOLUTION OF THE VARIOUS MANAGEMENT THEORIES 
A. INTRODUCTION 

The systems approach to manag^ent is one of the many 
theories of management that exist today. There are however 
some clearly identifiable concepts which provide a systematic 
look at these various theories. The following paragraphs 
[Luthans, 1973] and Figure 10 provide an aid in understanding 
these theories and their relationship to each other. 

The starting point of management theories can be traced to 
Henri Fayol. In 1916 he identified the universal functions 
of management as processes of planning, organizing, commanding, 
coordinating, and controlling. Fayol 's work did not however 
become the mainstream of management theory until the 1950' s. 

His process theories have remained relatively intact, however 
some of the terminology has changed. Fayol ' s "commanding" 
process is now known as "directing" . Dispite the changes in 
terminology and principles the universality assumption of the 
process approach is still made and remains as the theoretical 
base for management. 

In recent years the process approach has been overrun by 
other theoretical approaches. By 1960 two separate paths 
emerged from the process approach in opposite directions. These 
new paths became known as the quantitative and behavior ap- 
proaches to management. 
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FIGURE 10 

EVOLUTION OF THE VARIOUS MANAGEMENT THEORIES 
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B. APPROACHES TO MANAGEMENT 



1 • The Quantitative Approach 

The quantitative approach, emerging in early 1960, has 
its roots in the scientific management movement that actually 
predates the process approach. This approach was characterized 
by the techniques of operations research which used various 
mathematical models to solve decision problems. However it 
soon became apparent that this approach fell short of providing 
a theoretical base for management as a whole, although opera- 
tions research techniques did provide effective tools for 
management decision making. 

Starting in about 1970, the quantitative approach 
turned away from emphasis on narrow operation research tech- 
niques toward a broader perspective of management science. 

In addition to the quantitative decision making techniques 
and model building as in OR the function of computerized in- 
formation systems and operations management were incorporated. 
This then marked the return toward a more broadly based theory 
of management. 

2 . The Behavioral Approach 

The other separation from the process approach, the 
behavior approach, was charactirized by human relations. 
Assumptions were made about human beings and solutions to 
behavioral problems were offered. Improving morale was a 
major concern, which did no harm, but unfortunately not all 
desired results were obtained. 
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Again around 1970 the behavioral approach had a parallel 
development with the quantitative approach. A more broadly 
based organizational behavioral approach was followed. More 
direct attention is devoted to organization theory and organi- 
zational development. This approach is now the result of 
interaction between the human being and the formal organization. 

3 . The Systems Approach 

While the quantitative and behavioral approaches were 
going their separate ways the systems approach appeared. Since 
1960 this approach took up where the process approach left off 
in unifying management theory. 

The theoretical systems approach can be traced back 
to the natural and physical sciences of the early 1950 's. The 
application to management has been much more recent. The 
systems approach stresses the interrelatedness and inter- 
dependency of the parts to the whole. 

As has been noted, both the management sciences and 
organizational behavioral approaches are heading back toward 
the main path of systems. In management science, the new 
emphasis on computer applications and operations management 
techniques are systems based. On the other side the formal 
organization is viewed as a system consisting of structures, 
processes, and technology and is composed of the human system 
which contains a biological-physiological structure, psycho- 
logical processes, and a personality. 

Only time will tell if the systems approach will 
unitify the quantitative and behavioral approaches to management. 
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If the three approaches do actually merge in the future than 
something different from the sum of their parts may emerge. 

The future theory is now being called the contingency theory. 

4 . The Contingency Theory 

The contingency theory, sometimes referred to as situa- 
tional theory is starting to emerge from the many management 
theories. This theory has been advanced largely by practicing 
managers. The complex, changing environment around an organiza- 
tion has created the situation of "it all depends" type of 
management philosophy. 

The systems approach may however in time evolve to 
meet these situational theories. An open, as opposed to closed, 
systems view is better able to cope with increased complexities 
and environmental influences facing managers today. Systems 
concepts such as entropy and equif inability are quite applicable 
to the present managerial situations. 

Nevertheless a mid-range concept that falls between 
"simplistic, specific principles" and "complex, vague notions" 
is being called for by several notable authors in the area. 

This contingency approach "recognizes the complexity involved 
in managing modern organizations but uses patterns of relation- 
ships and/or configurations of subsystems in order to facilitate 
improved practices" [Kast, 1972] . This author, however, feels 
that flexibility in the systems approach can provide the de- 
sired results. 
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APPENDIX C 



SYSTEM MODELS 

Systems may be represented by various types of models. The 
model, as used in this thesis, is an abstract representation 
of real phenomena. The objective of the model is not to por- 
tray or identify all aspects of phenomena, but rather to point 
out the significant elements and the salient interrelations. 

The following types of models are listed in order of their 
degree of abstraction. 

1 . Verbal Models 

Verbal models rely on words to describe the elements 
and the interrelations of a particular system. Difficulties 
are encountered however in the ambiguities and semantic problems 
often encountered in verbal communication. 

2 . Schematic Models 

Schematic models are basically a picture of a system 
and are typically drawings or charts that present systems 
elements or their attributes together with a single relation. 
Static schematic models depict a set of elements and their 
relations at a given point in time; such as maps, bar charts, 
Gantt Charts, the breakeven chart, and the organizational 
chart. Flow schematic models are schematic models that protray 
flow or movement of some description among the system elements, 
such as the Program Evaluation and Review Technique (PERT) , 
and the decision tree. Dynamic system models are schematic 
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models that depict a transformation process. Inputs are pro- 
cessed into outputs under the direction of a controller. It 
is this model which will be used to analyze the systems approach 
to test and evaluation management. 

3 . Iconic Models 

Iconic models are static three-dimensional representa- 
tions of physical objects such as models used in wind tunnel 
testing . 

4 . Analog Models 

Analog models seek to describe the operational charac- 
teristics of other systems through the process of analogy. 

These models are based on the premise that the behavior of 
full-scale systems may be studied by examining the behavior 
of other systems with similiar characteristics. 

5 . Mathematical Models 

Mathematical models are symbolic models which represent 
the highest level of abstraction in systems model construction. 
These models are either deterministic or stochastic in nature 
and may be either descriptive or optimizing in purpose. 
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